<html>
<head><meta charset="utf-8"><title>cache misses · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html">cache misses</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="205852671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205852671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205852671">(Aug 03 2020 at 22:05)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> so I got into a discussion with <span class="user-mention" data-user-id="216025">@Siân Griffin</span> about measuring things and cache misses came up again. would be nice to have that kind of data as well (and I suspect it might be possible to collect it with my <code>rdpmc</code> infra as well)</p>



<a name="205852831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205852831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205852831">(Aug 03 2020 at 22:06)</a>:</h4>
<p>I think they'd indeed be helpful</p>



<a name="205852858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205852858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205852858">(Aug 03 2020 at 22:06)</a>:</h4>
<p>it should be fairly easy to add nowadays, someone just needs to do  it</p>



<a name="205852913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205852913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205852913">(Aug 03 2020 at 22:07)</a>:</h4>
<p>mostly within self-profile, I imagine, though maybe outside it as well</p>



<a name="205852931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205852931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205852931">(Aug 03 2020 at 22:07)</a>:</h4>
<p>the one source of non-determinism that I can think of is cache lines getting invalidated by context-switching (including from syscalls). which is basically the side-channel that we have several named vulnerabilities for now :P</p>



<a name="205853073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853073">(Aug 03 2020 at 22:09)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> tbh it would've been nice to already have the (whole-process) data for the past few months, given what I'm planning on doing, welp. maybe we can just start measuring it on perf.r-l.o through <code>perf stat</code>? I guess a filter is used and that's why we don't already have that data?</p>



<a name="205853098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853098">(Aug 03 2020 at 22:09)</a>:</h4>
<p>yeah <a href="https://github.com/rust-lang/rustc-perf/blob/master/collector/src/rustc-fake.rs#L49">https://github.com/rust-lang/rustc-perf/blob/master/collector/src/rustc-fake.rs#L49</a></p>



<a name="205853107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853107">(Aug 03 2020 at 22:09)</a>:</h4>
<p>basically someone should just update that in a PR I think</p>



<a name="205853127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853127">(Aug 03 2020 at 22:09)</a>:</h4>
<p>and I guess this table <a href="https://github.com/rust-lang/rustc-perf/blob/master/site/static/index.html#L48">https://github.com/rust-lang/rustc-perf/blob/master/site/static/index.html#L48</a></p>



<a name="205853267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853267">(Aug 03 2020 at 22:11)</a>:</h4>
<p>there aren't other places in the code which hardcode stuff like that?</p>



<a name="205853282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853282">(Aug 03 2020 at 22:11)</a>:</h4>
<p>I think no, we removed all of them (or tried to)</p>



<a name="205853283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853283">(Aug 03 2020 at 22:11)</a>:</h4>
<p><a href="https://gist.github.com/Mark-Simulacrum/151b4948d6bd1ab86b5c1d9b36ba64de">https://gist.github.com/Mark-Simulacrum/151b4948d6bd1ab86b5c1d9b36ba64de</a></p>



<a name="205853290"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853290" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853290">(Aug 03 2020 at 22:11)</a>:</h4>
<p>that's the list of things we can collect without reading cpu manuals</p>



<a name="205853302"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853302" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853302">(Aug 03 2020 at 22:11)</a>:</h4>
<p>heh</p>



<a name="205853391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853391">(Aug 03 2020 at 22:12)</a>:</h4>
<p>well I guess technically <a href="https://gist.github.com/Mark-Simulacrum/2146f7cf547b6a4cdf98c43340bb5706">https://gist.github.com/Mark-Simulacrum/2146f7cf547b6a4cdf98c43340bb5706</a>? that's with sudo</p>



<a name="205853468"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853468" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853468">(Aug 03 2020 at 22:13)</a>:</h4>
<p>I don't think that has extra Hardware events</p>



<a name="205853545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853545">(Aug 03 2020 at 22:14)</a>:</h4>
<p>but also, ugh, only L1. so not a great predictor for DRAM bandwidth</p>



<a name="205853660"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853660" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853660">(Aug 03 2020 at 22:15)</a>:</h4>
<p>yeah :/</p>
<p>maybe TLB misses are like 'worst case'</p>



<a name="205853666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853666">(Aug 03 2020 at 22:15)</a>:</h4>
<p>but I imagine those are quite low</p>



<a name="205853675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853675">(Aug 03 2020 at 22:15)</a>:</h4>
<p>we can probably get L2 and L3 if we go read the cpu manual</p>



<a name="205853695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853695">(Aug 03 2020 at 22:15)</a>:</h4>
<p>actually, one concern might be that this is all virtualized... somehow</p>



<a name="205853749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853749">(Aug 03 2020 at 22:16)</a>:</h4>
<p>instructions are relatively stable but I think caches are presumably less so?</p>



<a name="205853761"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853761" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853761">(Aug 03 2020 at 22:16)</a>:</h4>
<p>also there isn't just one counter so you'd have to read like at least two and aggregate them</p>



<a name="205853911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205853911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205853911">(Aug 03 2020 at 22:18)</a>:</h4>
<p>I would suspect non-determinism would mostly come from other address spaces, i.e. the context-switching I mentioned before (and memory allocation non-determinism, which I don't know the full difficulty of removing that, but that also impacts instruction counts through pointer hashing)</p>



<a name="205854367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205854367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205854367">(Aug 03 2020 at 22:24)</a>:</h4>
<p>well, we can at least try</p>



<a name="205854412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/205854412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#205854412">(Aug 03 2020 at 22:24)</a>:</h4>
<p>worst case the data is just super noisy and we ignore it</p>



<a name="206545491"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206545491" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206545491">(Aug 11 2020 at 07:03)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> If you want to improve measurement accuracy, you could always pin a process to a CPU, and enable full-no-hz. With that, you should get very low variance.</p>



<a name="206545516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206545516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206545516">(Aug 11 2020 at 07:03)</a>:</h4>
<p>In the absence of that, cache misses could vary more, but they're still useful.</p>



<a name="206545571"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206545571" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206545571">(Aug 11 2020 at 07:04)</a>:</h4>
<p>There are performance counters for it. Perf abstracts over the necessary ones for each CPU.</p>



<a name="206588122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206588122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206588122">(Aug 11 2020 at 15:17)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> how do you pin a process to a cpu? :0<br>
And does it also give you exclusive access to its cache for the duration of the process?</p>



<a name="206588156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206588156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206588156">(Aug 11 2020 at 15:17)</a>:</h4>
<p>(I'm curious; didn't know this was a thing)</p>



<a name="206589147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206589147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206589147">(Aug 11 2020 at 15:25)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> My impression was that the process pinning you describe requires us to dedicate N cores - currently rustc can use all 12 virtual cores on the collector but I think we'd need to limit to 11 with this, right?</p>



<a name="206589750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206589750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206589750">(Aug 11 2020 at 15:29)</a>:</h4>
<p>Note that we also have <code>perf stat</code> running  on all of our runs and such.</p>



<a name="206591440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206591440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206591440">(Aug 11 2020 at 15:44)</a>:</h4>
<p><span class="user-mention silent" data-user-id="212698">Félix Fischer</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/cache.20misses/near/206588122">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> how do you pin a process to a cpu? :0<br>
And does it also give you exclusive access to its cache for the duration of the process?</p>
</blockquote>
<p><code>taskset(1)</code></p>



<a name="206591633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206591633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206591633">(Aug 11 2020 at 15:45)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/cache.20misses/near/206589147">said</a>:</p>
<blockquote>
<p>Josh Triplett My impression was that the process pinning you describe requires us to dedicate N cores - currently rustc can use all 12 virtual cores on the collector but I think we'd need to limit to 11 with this, right?</p>
</blockquote>
<p>Not necessarily. core pinning forces process to be scheduled on the particular cores when its ready to run, but it does not mean other processes cannot be run on those cores.</p>



<a name="206591784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206591784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206591784">(Aug 11 2020 at 15:47)</a>:</h4>
<p>Preventing linux (at least) from scheduling anything on particular cores requires fiddling with kernel cli</p>



<a name="206592008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206592008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206592008">(Aug 11 2020 at 15:48)</a>:</h4>
<p>so what you'd have to do to ensure core is dedicated to a particular process is: 1. fiddle with kernel cli to ensure scheduler never schedules anything on that core normally; 2. probably adjust something to ensure same holds for IRQs; 3. taskset the interesting process to run on the core that has been dedicated for this through steps 1 and 2.</p>



<a name="206614483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206614483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206614483">(Aug 11 2020 at 18:44)</a>:</h4>
<p>yeah I think that's going to be pretty untenable from what it sounds like for perf specifically, because we can't afford to serialize (or anything close to it) dependency building</p>



<a name="206614545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206614545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206614545">(Aug 11 2020 at 18:45)</a>:</h4>
<p>maybe we can taskset a bunch of things onto the same set of cores though? I'd need to read up on how it all works out</p>



<a name="206617233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206617233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206617233">(Aug 11 2020 at 19:04)</a>:</h4>
<p>You could taskset just the crate being benchmarked. If it is an executable or pipelining is disabled, that rustc instance will be the only rustc instance when it runs.</p>



<a name="206617564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206617564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206617564">(Aug 11 2020 at 19:07)</a>:</h4>
<p>well we still need to build the dependencies, and if I understand <span class="user-mention" data-user-id="123586">@nagisa</span> right, if I configure the kernel to avoid (say) vcores 0-10 (out of 12) in normal functioning then I need to <em>always</em> taskset every process onto those cores if I want parallelism in dependency building</p>



<a name="206620744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206620744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206620744">(Aug 11 2020 at 19:35)</a>:</h4>
<p>all this sounds doable on <a href="http://build.lyken.rs">build.lyken.rs</a> which is 24C/48T (although we'd have to set it up with our sysadmin), so all of these details seem helpful, thanks!</p>



<a name="206620779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206620779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206620779">(Aug 11 2020 at 19:35)</a>:</h4>
<p>another thing is that most of the useful data, IMO, comes from the pre-LLVM part of the compilation, which can be kept single-threaded</p>



<a name="206620867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206620867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206620867">(Aug 11 2020 at 19:36)</a>:</h4>
<p>that is, I'd only measure check-only builds, in terms of optimizing rustc itself</p>



<a name="206627542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206627542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206627542">(Aug 11 2020 at 20:34)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/cache.20misses/near/206617564">said</a>:</p>
<blockquote>
<p>well we still need to build the dependencies, and if I understand <span class="user-mention silent" data-user-id="123586">nagisa</span> right, if I configure the kernel to avoid (say) vcores 0-10 (out of 12) in normal functioning then I need to <em>always</em> taskset every process onto those cores if I want parallelism in dependency building</p>
</blockquote>
<p>You could configure the kernel to avoid some cores, then use taskset to also allow the rustc of dependencies on those cores and finally use taskset to limit the benchmarked crate to those cores once all dependencies are compiled.</p>



<a name="206627649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206627649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206627649">(Aug 11 2020 at 20:35)</a>:</h4>
<p>yeah I'm not enthusiastic about "taskset to also allow the rustc of dependencies on those cores" -- I presume I can't just taskset cargo, right? I need to do so for every process (or thread?)</p>



<a name="206627898"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206627898" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206627898">(Aug 11 2020 at 20:37)</a>:</h4>
<p>We are already using a rustc wrapper I think. That one could spawn rustc through taskset.</p>



<a name="206627981"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206627981" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206627981">(Aug 11 2020 at 20:38)</a>:</h4>
<p>right, but if you need to do it for each process it won't cover build scripts etc which is pretty bad</p>



<a name="206628192"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206628192" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206628192">(Aug 11 2020 at 20:39)</a>:</h4>
<p>Build scripts are generally not using multiple threads, right? So it should be fine to keep running them one the one or more cores not reserved for benchmarking.</p>



<a name="206628302"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206628302" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206628302">(Aug 11 2020 at 20:40)</a>:</h4>
<p>they definitely are, for spawning c/c++ compilers</p>



<a name="206628638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206628638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206628638">(Aug 11 2020 at 20:42)</a>:</h4>
<p>Ah, yes. More conplex but maybe add a gcc wrapper? The build scripts should look at the CC env var, right?</p>



<a name="206628647"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206628647" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206628647">(Aug 11 2020 at 20:42)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/cache.20misses/near/206627649">said</a>:</p>
<blockquote>
<p>I presume I can't just taskset cargo, right?</p>
</blockquote>
<p>Why not? AFAIK it would be inherited to child processes</p>



<a name="206628775"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206628775" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206628775">(Aug 11 2020 at 20:44)</a>:</h4>
<p>in fact I was using it that way in <a href="https://github.com/rust-lang/cargo/issues/8595">cargo#8595</a></p>



<a name="206629073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206629073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206629073">(Aug 11 2020 at 20:46)</a>:</h4>
<p>Cargo processes the rustc output, which would cause a bit of noise when run on the same core as rustc.</p>



<a name="206629117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206629117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206629117">(Aug 11 2020 at 20:46)</a>:</h4>
<p>The noise may be low enough though.</p>



<a name="206629266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/cache%20misses/near/206629266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/cache.20misses.html#206629266">(Aug 11 2020 at 20:47)</a>:</h4>
<p>well, we can plausibly do something like this:</p>
<ul>
<li>by default, i.e., when building dependencies, run taskset with all cores allowed on cargo -- this should remove any harmful perf effects</li>
<li>in the final compilation, when we're benchmarking rustc performance, we know that rustc will only get invoked once, so we can <em>not</em> taskset cargo and instead taskset the rustc that'll get spawned</li>
</ul>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>